Although a custom coordinate system cannot be published to FME Server, there are two ways they can be made available to the FME Server:
Modify the MyCoordSysDefs.fmw file:
Add its definition to the following file where you have an FME Engine installed:
<FMEServerDir>\Server\fme\Reproject\MyCoordSysDefs.fme
MyCoordSysDefs.fme is a file that contains custom coordinate systems defined by the user.
Add it to the the CoordinateSystems subdirectory of a shared resource directory. For more information, see To make a custom resource available using the shared resource directive:.